For any given problem, relevant data is often available from a wide variety of data sources. Business Intelligence (BI) software applications take such disparate data, and make it available in a more helpful and uniform form in a data warehouse. Extract, Transform, and Load (ETL) techniques are used in the industry to move such heterogenic data into a data warehouse from which useful information may be more easily acquired by business logic of the business intelligence. The extract phase involves acquiring data from the data sources in their native form. The transform phase involves transforming such data into a form that is more understandable to the business logic. The load phase involves making such transformed data available to the business logic.
Typically, ETL processing techniques process once or a few times in a day on a small number of data sources. Furthermore, the ETL processing the sequencing of the ETL steps is often hard-coded. For example, to account for dependencies between data sources, perhaps the ETL process waits until all database backups for the day have arrived and been made available to the ETL process.